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ABSTRACT 

In an effort to determine success in fostering 
student development as well-rounded human beings, Northwestern 
Michigan College (NMC) undertook a study of student achievement 
related to significant learning outcomes established by the college. 
The study focused on four measures of achievement: results from a 
spring 1995 administration of the Collegiate Assessment of Academic 
Proficiency (CAAP) , results from a spring 1995 American College 
Testing College Outcomes Survey (COS) report, an NMC plan for 
assessing academic achievement, and outcomes from an Instructional 
Development and Effectiveness Assessment (IDEA) conducted from 1990 
to 1995. Study findings included the following: (1) on the CAAP, NMC 
students scored above the national mean in each of four academic 
skills (i.e., writing, mathematics, reading, and critical thinking); 
(2) regarding the COS, the learning outcome ranked highest by 
students was acquiring knowledge and skills needed for a career, 
while this outcome was also ranked highest in terms of 



i» tudent s 
will be 
to the IDEA, 
(i.e. , gaining 



self-assessment of progress; (3) NMC ' s assessment plan 
implemented over the coming year; and (4) with respect 
students consistently rated their learning in 10 areas 
factual knowledge, learning principles and theories, developing 
professional skills, learning a discipline's methodology, improving 
thinking and problem solving skills, developing creative capacities 
and effective communication skills, gaining a general, liberal 
education, developing personal responsibility, and furthering 
self-understanding) higher than the system-wide norm. Results from 
the COS and CAAP are appended. (MAB) 
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NORTHWESTERN MICHIGAN COLLEGE 



INDICATORS OF SUCCESS 
SIGNIFICANT LEARNING OUTCOMES 
Annual Progress Report 

SEPTEMBER 1995 
Significant Learning Outcomes Policy 

Northwestern Michigan College will help students realize their full potential and will 
foster their development as well-rounded human beings who are able to think independently, 
to communicate clearly, to speculate intelligently, to access and generate knowledge, and to 
channel that knowledge to enhance the public good 

Purpose of Report 

This report provides recent evidence of how well NMC students are realizing the 
qualities spelled out in the policy above. It is therefore an indication of how well our 
instructional efforts and student support services are achieving their goals. This report also 
provides an update on the planning activities of the Significant Learning Outcomes Team and 
other groups within Educational and Student Services. 

Because this is the first annual report under the Board Ends Policy provisions, it will be 
almost entirely focused on the evidence currently available for assessing learning outcomes. It 
will describe other means for taking stock that are now under development, but only in general, 
tentative terms. And it will postpone a discussion of specific strategies for improvement until the 
Team has had time for additional training, conferencing with other teams, and prioritizing the 
areas where our efforts should be concentrated. 

The four areas of measurement that will serve as indicators of Significant Learning Outcomes are: 

1 . Collegiate Assessment of Academic Proficiency (GAAP) Report 

2. American College Testing College Outcomes Survey Report 

3. North Central Association- Approved Student Outcomes Assessment Report 

4. IDEA Learning Outcomes Summary 

It should also be noted that evidence of significant learning while at NMC is also provided by 
findings to be reported by the Academic Transfer Team and the Successful Student Placement 
Team. 

1 
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M.,.nr.n..nt Area i • r.^^.o^.ie Assessment of Acdemic Proficiency (CAAP) Repor t 

NMC was involved in the original development and field testing of this ACT assessment tool and 
has since continued to refine methods of sampling studems and administering the academic 
proficiency tests most closely linked with our educational mission statements. 

Conducted on an annual basis, this assessmem measure requires that students with over thirty 
semester hours schedule a session for a test in one of the following skills: Wntmg, Mathematics, 
Reading, or Critical Thinking. Studems are given one of the four tests randomly, which means 
that approximately one quarter of the emire group will be tested in any one academic skill. The 
results of these tests are returned to each studem, used to compile an NMC summary report, and 
figured imo the computation of national means. For three of the four skills (Critical Thinking is 
excluded), comparisons are made with studems' skill levels when emering NMC, as indicated on 
their ASSET tests. 

Plans are set to include the GAAP results on students' permanem transcripts. However, the 
results do not affect graduation status or academic standing of individual studems. The dominant 
emphasis in this testing program is on institutional effectiveness, but the results may also help 
students redirect their studies to strengthen weaknesses. 

Current Status ^ ,. , 

The results for the most recem GAAP assessmem (Spring 1995) reflect a samphng of 256 
studems from a cohort of approximately 325 individuals who met two criteria, over thirty hours 
of NMC credit and ASSET scores from at least six months prior. The national means hsted 
below also show the total number of community college studems tested nationwide. 

Percentile Means 

NMC National 
Writing Skills 64.0 62.4 

Mathematics 59.0 56.2 

_co n=14,144 

Reading 61.6 61.0 

Critical Thinking 62.2 61.1 

„=67 "=12'^«^ 
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Note: At the time of this report, the 1995 ASSET-CAAP longitudinal comparisons have not 
been processed by ACT and are therefore unavailable. However, this comparison data is 
available in a copy of a Student Outcomes Assessment memo included as the second document in 
the appendix. Also, the national means for ASSET scores, though on record with ACT, are not 
included in this report but will be used in helping to determine NMC's standards of performance 
for this indicator. 

Conclusions: 

As in nearly all past CAAP summaries, NMC students have scored above the national mean in 
each of the four academic skills. This is encouraging evidence of the relative strength of NMC's 
student performance. However, plans must be developed for increasing the sample size for each 
skill test and determining standards by which to gauge future success. In addition, the team will 
establish an appropriate time interval for making a series of ASSET-CAAP comparisons to 
analyze trends. Also, the team will use national means for ASSET to help adjust the gains made 
between ASSET and CAAP in these comparisons. 

Measurement Area 2: ACT College Outcomes Survey Report 

Once a year, the same group of students selected for the CAAP testing program complete a 
detailed survey that rates their progress toward various educational goals while at NMC. Most 
recently, for example, in Spring 1995, this meant that 238 of the 256 students who took the 
CAAP test completed the Outcomes Survey. In this survey, how important each goal is to 
students and how much they believe their NMC experience contributed to their progress toward 
this goal are measured and compared. 

Though not a direct measure of academic skill or occupational competence, this self-assessment 
instrument provides one of the most telling indicators of students' growih after a year or more of 
NMC study. It also helps round out the "multiple measures" approach to assessment, as 
mandated by NCA. Because it is an annual assessment activity, the Indicator Team can track 
these outcomes over time and monitor trends. 

Current Status 

The outcomes from the Spring 1995 survey are summarized here: 

Importance of and Progress Toward Attaining Outcomes at NMC 

The six most important learning outcomes, as ranked by the 238 students surveyed, are listed on 
the next page, followed by how students ranked them in terms of the progress they felt they had 
made towards these outcomes while at NMC The ftjll list of 26 outcomes and their rankings are 
included in the appendix. 
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Importance Rank Learning Outcome 



Progress Rank 



1 
2 
3 
4 
5 
6 



Acquiring knowledge and skills needed for career 

Becoming competent in my major 

Learning to think and reason 

Developing problem-solving skills 

Listening to and understanding what others say 

Effectively using technology (computers, etc.) 



1 
3 
2 
5 
8 
13 



The comparison of importance rank and progress rank for the all 26 learning outcomes is 
analyzed more completely in the full ACT Outcomes Survey Report.(See also the appendix.) 

Conclusions 

With the exception of the outcome ranked sixth for importance (Effectively using technology), 
there is very close correspondence between how important students think these outcomes are 
and how much they felt NMC contributed to their progress toward them. Because students feel 
that knowing how to use technology is an important learning goal, and because it ranked 
thirteenth among 26 items in progress, this learning objective needs to be addressed when the 
team develops its standards and strategies for improvement. 

Measurement Area 3: North Central Association-Approved Outc omes Report 

After over four years of study, the Student Outcomes Assessment Committee submitted its plan 
for assessing student academic achievement to NMC's Board of Trustees (February 1995) and to 
the North Central Association's Commission on Higher Education (March 1995). North Central 
approved the plan in June of this year, assuring NMC's top accreditation status through the year 
2000. 

This comprehensive plan (See the separate document, "Signs and Symbols: Assessment 
Strategies at Northwestern Michigan College") includes both the measures described in this 
report and many others that are conducted at the divisional level across the disciplines. These 
range from pre- and post-testing in individual courses, to committee review of students' formal 
written work, portfolio assessment, state and national certification results, and division-specific 
student questionnaires. Unlike the nationally normed instruments that provide external 
comparisons, these efforts aim more directly at what students have done in meeting Core and 
program requirements. 
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Current Status 

The academic divisions are in various stages of completing and implementing their assessment 
strategies. Although some divisions have measurement data as part of ongoing annual 
assessment, the Indicator Team has not yet had time to begin the process of selecting and 
compiling this information in its first annual report. Planning this process and prioritizing 
strategies based on the findings will be the major activity of the team over the next year. 

Conclusions 

No conclusions have been made for this report. 

Measurement Area 4: Instructional Development and Effectiveness Assessment (IDEA) 

Learning Summary 

When individual courses are evaluated each year, students complete ten items that rate their level 
of learning in the course. Because IDEA is course-specific, its primary use is for evaluating 
faculty performance and improving instruction course by course. It is therefore a complement to 
the annual Student Outcomes Survey described above. But because the individual course data is 
pooled and normed both locally and throughout the nation among colleges using IDEA, it also 
provides a sound method for measuring NMC's educational effectiveness— as perceived by 
students while still immersed in the process of learning. 

NMC has been conducting this assessment measure for over ten years, making it an excellent 
indicator of sustained success (or need for improvement). 

Current Status 

The most recent IDEA measurement was taken Spring Semester, 1995. On the next p^.ges, a 
series of ten charts (one for each learning outcome as indicated in the chart title) sho\v 5 this 
spring's results and those for the previous six semesters or terms. Each chart indicates the level 
of the system wide means and NMC's means on a five-point scale, with five being strong 
favorable rating by students in the course, 1 being the lowest rating. 



Subject Matter Mastery 

Gaining Factual Knowledge - Essential or Important 
[Gaining factual knowledge (terminology, methods, classification, trends).] 




Subject Mastery 

tearing Principles & Theories - Essential or Important 
[Learning fundamental principles, theories, or generalizations.] 




--^IDEA(data base) 
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Subject Matter Mastery 

Developing Professional Skills - Essentia! or Important 
[Developing specific skills, points of view, and competencies needed by professionals in the field.] 




Subject Matter Mastery 

Learning the Process of the Discipline's Method - Essential or Important 
[Learning how professionals in this field go about the process, of gaining knowledge.] 




Development of General Skills 

Improving Thinking & Problem Solving Skills • Essential or Important 
(Learning to apply course material to improve decision making, rational thinking, problem solving ] 




Developnnent of General Skills 

Developing Creative Capacities - Essential or Important 




Development of General Skills 

Developing Effective Communication Skills - Essential or Important 
[Developing skill in expressing myself orally or in writing.] 




Personal Development 

Gaining a General Liberal Education - Essential or Important 
[Gaining a broader unerstanding and appreciation of intellecutal-cultural activity.] 




Personal Development 

Developing Personal Resposibility - Essential or Important 
(Developing a sense of personal responsibility (self-reliance, self-discipline).] 



-«-NMC 

— IDEA (data base) 



Personal Development 



Discovering Implication for Self Understanding - Essential or Important 
[Discovering the implications of the course for understanding myself (interests, values, talents)] 



1 ' 
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-^IDEA (data base) 



^0 13 
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Conclusions 

It is obvious that students consistently rate their learning in these ten areas higher than the 
system-wide norm. In some cases the NMC margin over the system norm is truly impressive 
(margins of .4 or .5 on a 5-point scale clearly representing strong performance). In a few cases, 
NMC matched the system-wide mean, and in only two cases did NMC fall below the system- 
wide mean-Developing Creative Capacities in Fall 1992 and Winter 1994. Of greatest 
significance in watching for evidence of improvement, all but one of the Spring 1995 outcomes 
came in higher for NMC than the system- wide means, and that single exception matched the 
system mean for the outcome "Learning the process of a discipline's methods." 

The task for this Indicator Team is to establish meaningful standards for the IDEA measure, then 
analyze recent trends to prioritize strategies for making improvements if and where they are 
needed. This will be included in the Team's work for 1995-96 

NMC Standards of Performance 

As explained in the conclusions for each measurement area, the Significant Learning Outcomes 
Team will use the coming year to determine what standards NMC should meet. Because several 
of the indicators of student learning are numerical and are norm-referenced, setting a standard is 
an extremely sensitive process, if the standard is to be valid and realistic. 

Strateeies for Improvement 

Strategies for improvement will be developed following the determination of standards. The 
Team expects to have at least three of the standards and preliminary strategies on paper by 
December 1995. 



Prepared by Jack Berman, Da\id Donovan, Dianne Keelan, and John Pahl 
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Appendix 

Item 1 : Twenty-six Student Outcomes 
ACT Survey Results, 1995 

Item 2: ASSET-CAAP Comparisons 
Assessment Committee Memo, Nov. 1994 
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TO. Student Outcomes Assessment Committee 

FROM: John Pahl 

DATE: Nov. 10, 1994 

SUBJ: Summary of most recent GAAP results and related thoughts 



In June, ACT sent a thick packet of information following our most recent 
administration of CAAP. For the first time, the results included a longitudinal 
analysis, called ASSET-CAAP Institutional Cohort Report, as well as the CAAP 
scores and the student opinion Survey (a self-assessment inventory). 

The longitudinal document is in a new reporting format, one that is fairly easy to 
interpret but only after the three pages of instructions are digested. To reduce any 
further demand on your time, I've pulled out some salient points and added a few 
impressions. Dave Donovan has also been through these same pages and will 
probably have some further ideas to add. Naturally, you may want to look at the 
report itself, including the 200 or so pages that go through the same results in a very 
finely textured fashion, looking at declared programs, age groups, etc. 

General Results of CAAP 

1. NMC students are slightly above the National mean in Writing, Critical 
Thinking, and Math, but slightly below in Reading. (360 students tested) 



NMC Nat'l Mean 

Writing 63.39 62.2 

Critical Thinking 61.91 61.4 

Mathematics 58.51 56.2 

Reading 60.09 61.2 



One year's results aren't enough for reaching conclusions. We need to get 
a baseUne established. Then watch for trends. 

2. The 1994 test group, significantly larger, scored slighlty lower than the 
1993 group, except in mathematics, which rose slightly. 



1994 1222 

Writing 63.39 64.02 

Critical Thinking 61.91 63.63 

Mathematics 58.51 57.89 

Reading 60.09 62.51 



The declines and increase may be a result of using a larger, more accurate 
group (360 vs. 270). In any case, the same principle appUes: we need a 
Q set of years to see a pattern and to see whether changes of one or two points 

ERIC are significant. 
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Longitudinal Analysis Results 

L Mathematics 

A total of 65% of those tested maintained their percentile ranking or 
improved it, (29% maintained level of performance; 36% improved,) 
Thirty-five percent performed at a lower leveL 

Given that there was no general math requirement in place over the 
years covered by this longitudinal comparison, the fact that 35% 
actually went down one or more deciles (10% levels) should not be 
surprising. The number staying at or improving their levels is 
encouraging. (We should remember that maintaining a percentile 
level can still mean improving in actual performance, since presxmiably 
the entire national cohort has improved as welL) After the CORE 
requirement is in place for several years, we'll be able to make an interesting 
comparison. 

Among originally weaker students (lower than mean on ASSET), 65% 
increased their standing in decile level, and of those, 36% rose above the 
mean on the CAAP. 

2. Reading 

A total of 60% of those tested maintained their percentile ranking or 
improved it. (16% maintained level of performance; 44% improved.) 
Forty percent performed at a lower leveL 

We can't factor in motivation accurately until ACT provides the further 
analysis it has underway (using the students' indication of effort on 
CAAP). But a good guess for decline in reading scores might be motivation 
when taking CAAP. Since reading is required across the curriculum, and one 
would assume it might improve even if we weren't trying to work expressly 
on it, it's hard to believe that 40% of those tested actually read less well 
relative to a national norm than they did before two years at NMC. 
Obviously we have to be careful in drawing such conclusions (or we'll 
beg the entire question of testing in the first place). But the motivational 
issue can be addressed in two ways--ACT's analysis factoring in students' 
indication of effort and our own ingenuity in increasing their motivation, 
a process already begun this spring. 

3. Writing 

A total of 71% of those tested maintained their percentile ranking or 
improved it. (23% maintained level of performance; 48% improved.) 
Twenty-nine percent performed at a lower level. 



The same cautions about motivation apply here (and for all of the scores 
from this last testing session). We should also emphasize that though 
ACT calls this "Writing," it's really editing of other people's writing. 
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Characteristics of the NMC Group 

1. Cohort Group ^, , 

This testing session had a well defined cohort group, selected by credit hours 
earned and time frame for having taken ASSET. Unlike other groups m the 
national norm, we had a very high percent (91%) who began NMC as 
freshmen, thus increasing the accuracy (motivation at GAAP time aside) in 
determining our effect on their longitudinal changes. 



2. Age 

Largest categories: (no National norms available.) 

19- 20, 32.8% 

20- 25, 25.3% 
26-30, 11.1% 
31-39, 17.5% 

3. Gender .ac 

Our female-male split is more balanced than the national norm, 56%-44 7c. 

4. Full- and Part-time n-:>c 
We had fewer full-time students tested, but not by much~NMC 73 /c; 

National 78%. 

5. Future Plans „ ^ 

Fewer of our students indicated transfer (30.8% vs. 46.6% nationally). 
More of our students indicated they would be returning to NMC 51.7% 
vs. 30% nationally. 

6. Field of Study 

The six largest groups were: 

Health professions 19.2% 
Business 14.4% 
Engineering 9.7% 
Education 9.4% 
Social Sciences 6.9% 
Trade/Industrial 5.3% 

The Outcome Survey ^ „ u 

The students' opinions on what was most important to them educationally, ho^^ 
much they felt they improved while at NMC, and how much they felt the College 
contributed to those gains are very interesting and helpful. The full report should 
be reviewed by all whose programs are indirectly involved and by anyone involved 
in institutional effectiveness. 

The following page provides a synopsis of just a few major summaries: 
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Items of most importance to students 

Students ranked the following outcomes as the top six in what they hoped to attain 
while at NMC. Students then ranked them among the top seven in assessing their 
progress in these outcomes while at NMC (There were twenty-six outcomes in all.) 

1. Acquiring knowledge and skills needed for a career 

2. Becoming competent in my major 
3- Learning to think in my major 

4. Developing problem-solving skills 

5. Listening to and understanding what others say 

6. Thinking objectively about beliefs, attitudes, and values 

The top twelve items ranked by importance as outcomes (the six above plus the six 
that follow) were ranked in the top sixteen as progress in outcomes while at NMC 

7. Drawing conclusions after weighing evidence, facts, and ideas. 

8. Speaking more effectively 

9. Effectively using technology (e.g., computers, hi-tech equipment) 

10. Further developing my study skills 

11. Developing openness to new ideas and practices 

12. Developing my creativity, generating new ideas and products 

Implications for CORE 

Students were generally favorable (3.40 -3.84 on a 5-point scale) in rating the positive 
effects on the required courses they took outside their majors. There were no 
negative scores on these items. (See page 10 of the Outcomes Survey.) 
Such courses helped students "think about their majors in the context of a larger 
wold view," or "build a framework to organize my learning within and across areas 
of study," or "become a more independent and self-directed learner," this last 
earning the highest score. This may strengthen our belief in the wisdom of our 
CORE decisions. (See also the note below about "Math and statistics" growth.) 

Personal Growth Assessment 

These summaries are worth looking at but are not listed in my summary because of 
their detail. Here students assess the contribution made by NMC, inside and outside 
of class. Personal growth items include such things as "becoming academically 
competent," "clarifying my personal values," and "increasing my intellectual 
curiosity." Ratings in these summaries were clearly favorable. 

Only Three Exceptions 

The only personal growth items to receive less than a 3.0 rating were: 
developing job seeking skills 

understanding math and statistics (not a requirement when students 
were tested, i.e., before CORE began) 

learning principles for improving global environment. 

cc: Other Student Outcomes Assessment Comm. members, Shelley Merrill 
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Northwesteml Michigan 
1 College 



Northwestern Michigan College is a comprehensive community college 
which is committed to open access, excellence in teaching and learning, 
and support for student success through career, enrichment, and transfer 
programs— in partnership with the communities it serves. 



» 



« 



Board of Trustees 
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Michael A. McManus, Treasurer 
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